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ABSTRACT 

A  methodology  to  measure  the  detection  and  classification  performance  of  a 
minehimting  sonar  is  outlined.  A  technique  for  specifying  the  detection  and 
classification  performance  in  a  contract  and  then  relating  that  to  the  performance 
measured  in  sea  trials  is  described.  The  report  also  shows  how  a  model  can  be  used  to 
adjust  the  required  performance  to  allow  for  the  effects  of  the  envirorunent. 
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Evaluation  of  the  Performance  of  a 
Minehunting  Sonar 


Executive  Summary 

The  performance  of  mine  htmting  sonars  fluctuates  with  the  environment,  with  target 
characteristics  and  with  operator  skill.  Because  of  the  variability  the  sonar  performance 
cannot  be  reliably  evaluated  by  a  single  measurement  but  must  be  determined  as  a 
statistical  sampling  problem. 

The  detection  performance  can  be  assessed  by  conducting  many  detection  runs  at 
various  target  aspects  and  generating  probability  of  detection  curves  and  confidence 
limits  from  the  restilts.  These  data  can  then  be  compared  to  the  specified  performance. 

In  practice  the  environmental  conditions  inevitably  deviate  from  the  ideal,  are  never  as 
specified  and  usually  degrade  sonar  performance.  The  influence  of  the  environment 
can  be  allowed  for  by  using  a  sonar  performance  prediction  model,  which  takes  into 
account  the  prevailing  conditions,  to  scale  the  specified  detection  range  and  using  the 
range  so  scaled  as  the  range  to  be  demonstrated. 

In  a  similar  sampling  process  the  classification  performance  of  a  minehunting  sonar 
can  be  evaluated  by  determining  probabilities  of  correct  classification  against  a  range 
of  mine  like  and  non-mine  objects. 
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1.  Introduction 


This  report  summarises  three  aspects  of  minehuntmg  sonar  performance: 

a)  It  describes  a  trials  methodology  to  measure  detection  and  classification 
performance. 

b)  It  shows  how  the  detection  and  classification  performance  can  be  specified  in  a 
contract  and  related  to  the  performance  measured  in  (a). 

c)  It  shows  how  a  model  of  the  sonar  can  be  used  to  allow  for  the  various 
envirorunents  encountered  during  sea  trials. 

Detection  performance  is  measured  by  the  probability  of  detection  (Pd)  as  a  function  of 
range.  Classification  performance  is  determined  by  measuring  two  separate 
probabilities,  the  probability  of  correctly  classifying  a  mine  as  a  mine  and  the 
probability  of  correcdy  classifying  a  non-mine  as  a  non-mine.  The  report  shows  how 
confidence  hmits  for  each  of  the  measured  probabihties  can  be  obtained.  The 
classification  performance  is  measured  by  requiring  the  operator  to  classify  a  series  of 
objects  either  as  mines  or  non-mines. 

The  methodology  described  here  was  used  successfully  during  trials  of  RAN 
minehunters.  These  procedures  have  been  jointly  developed  by  the  RAN  and  the 
Maritime  Operations  Division  of  DSTO. 


2.  Factors  Affecting  Detection  Performance 

During  trials  and  exercises  it  is  apparent  that  the  detection  performance  of  a 
minehxmting  sonar  fluctuates  from  ping-to-ping.  Variations  are  ^o  observed  over 
longer  periods  and  these  variations  occur  even  when  the  environment  appears  to  be 
constant.  Because  the  sonar  performance  fluctuates,  the  measurement  of  performance 
must  be  treated  as  a  statistical  sampUng  process.  It  is  not  sufficient  to  carry  out  a 
single  detection  run  against  a  target  and  to  use  the  detection  range  as  a  reliable 
indicator  of  sonar  detection  performance.  The  detection  performance  of  a 
minehunting  sonar  is  dependent  on  several  factors,  each  of  which  fluctuates 
independently.  These  factors  include: 

1.  Bottom  Reverberation.  The  area  of  the  bottom  that  is  generating  reverberation  will 
change  from  ping-to-ping.  This  will  cause  the  level  of  the  reverberation 
background  to  vary.  Similar  considerations  will  apply  for  surface  and  volume 
reverberation. 

2.  Noise.  The  noise  level  against  which  the  echo  must  be  detected  may  depend  either 
on  the  ship's  own  self  noise  due  to  hydrodynamic  flow  or  machinery  noise,  or  on 
ambient  noise  in  the  ocean.  These  noise  sources  are  independent  and  will  vary 
with  time. 
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3.  Propagation  Loss.  Propagation  loss  between  the  sonar  transducer  and  the  mine 
will  vary  due  to  microstructure  in  the  water.  As  the  mine-htmter  moves  through 
the  water  the  target  will  be  insonified  by  slightly  different  transmission  paths.  This 
wiU  cause  fluctuations  in  propagation  from  ping-to-ping. 

4.  Target  strength.  The  target  strength  of  a  typical  mine  varies  with  aspect  over  a 
dynamic  range  of  approximately  20dB,  and  variations  of  5  to  10  dB  over  aspect 
changes  of  oidy  a  few  degrees  are  typical.  As  the  mine-htmter  closes  the  target  the 
aspect  will  usually  vary  slightly  from  ping-to-ping  and  the  target  strength  will 
fluctuate. 

5.  Operator.  The  ability  of  the  operator  to  detect  targets  ( i.e.  the  detection  threshold) 
will  vary  with  time  due  to  the  state  of  alertness,  fatigue  and  sea  sickness.  There 
will  also  be  variation  between  operators  due  to  different  skill  levels. 

By  taking  appropriate  steps  the  combined  effect  of  each  of  these  variations  can  be 

minimised,  but  not  entirely  eliminated. 


3.  Specification  of  Detection  Performance 

The  measure  of  detection  performance  used  is  that  of  probability  of  detection  as  a 
frmction  of  range.  This  is  defined  as  the  probability  of  detection  by  the  time  that  the 
minehxmter  has  closed  the  range  from  infinity  to  the  given  range.  This  statement  of 
detection  performance  is  only  valid  for  one  particular  environment  and  target.  The 
state  of  operator  alertness  must  also  be  stated. 


4.  Measurement  of  Probability  of  Detection 

The  method  of  measuring  time  probability  of  detection  as  a  function  of  range  is  now 
described.  On  each  nm  the  minehunter  commences  at  a  sufficient  range  to  be  out  of 
contact  with  the  mine.  The  range  is  closed  xmtil  a  detection  is  made  by  the  operator, 
the  range  is  noted  and  the  minehunter  then  breaks  off  contact  and  opens  the  range  for 
another  run.  The  detection  range  is  measured  for  a  number  of  approaches  to  the  mine. 
Several  types  of  detection  can  be  made;  if  the  position  of  the  mine  is  known  to  the 
operator  the  target  is  said  to  be  indicated.  This  will  be  the  best  detection  performance 
that  can  be  expected  of  the  sonar.  If  the  operator  is  tmaware  of  the  position  of  the  mine 
the  target  is  said  to  be  unindicated.  This  is  closer  to  the  operational  situation  but  in  all 
such  controlled  trials  there  is  a  reasonable  degree  of  operator  alertness.  In  an 
evaluation  it  is  recommended  to  proceed  from  the  first  stage  in  which  indicated  targets 
are  used  to  the  second  stage  using  unindicated  targets.  The  final  stage  should  be  a  free 
play  exercise  in  which  the  minehunter  is  tasked  to  clear  a  given  area  of  mines.  This 
will  give  a  measure  of  the  area  clearance  rate  which  is  an  important  operational 
measure  of  effectiveness. 
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Consider  now  the  measurement  of  probability  of  detection  versus  range  for  indicated 
detections  of  a  single  mine.  As  an  example,  20  detection  runs  are  carried  out  over  a 
period  of  a  few  hours,  the  environment  is  monitored  and  determined  to  be  constant, 
thus  the  measured  detection  performance  will  consist  of  the  set  of  20  ranges.  Table  1 
shows  a  typical  set  of  detection  ranges.  It  is  quite  common  to  find  a  variation  of  a 
factor  of  2  between  the  maximum  and  minimum  ranges.  If  no  detection  is  made  during 
a  run,  zero  range  should  be  logged.  In  order  to  calculate  the  probability  as  a  function  of 
range,  the  ranges  are  ranked,  with  the  shortest  range  at  the  top  of  the  column.  A 
probability  of  1/n,  where  n  is  the  number  of  runs  carried  out,  is  assigned  to  the 
detection  at  the  longest  range.  In  this  case  since  n=20  a  probability  of  1/20=0.05  is 
assigned  to  the  detection  at  the  longest  range.  For  each  successive  reduction  in 
detection  range  in  the  column  the  Pd  in  incremented  by  0.05.  Thus  the  Pd  for  the 
shortest  range  detection  is  equal  to  1.0.  For  the  example  given,  Pd=0.05  at  a  range  of 
507  m  and  Pd=l  at  a  range  of  157  m,  ie  aU  mines  wiU  have  been  detected  by  the  time 
die  range  closes  to  157  m. 


Table  1.  Typical  Detection  Ranges  and  Estimated  Probabilities  of  Detection. 


Run 

Number 

Detection 

Range 

(m) 

■ 

Rariked 

Detection 

Ranges 

Probability 
of  Detection 

90%  Confidence  Limits 
Lower  Upper 

1 

319 

157 

1.00 

0.861 

1.000 

2 

253 

217 

0.95 

0.784 

0.997 

3 

270 

230 

0.90 

0.717 

0.982 

4 

312 

253 

0.656 

0.958 

5 

359 

270 

0.80 

0.599 

0.929 

6 

157 

298 

0.75 

0.544 

0.896 

7 

217 

312 

0.70 

0.492 

0.860 

8 

230 

318 

0.65 

0.442 

0.823 

9 

351 

319 

0.60 

0.394 

0.783 

10 

450 

328 

0.55 

0.347 

0.741 

11 

431 

341 

0.50 

0.302 

0.698 

12 

507 

351 

0.45 

0.259 

0.653 

13 

389 

359 

0.40 

0.217 

0.606 

14 

421 

378 

0.35 

0.177 

0.558 

15 

378 

389 

0.30 

0.140 

0.508 

16 

409 

409 

0.25 

0.104 

0.456 

17 

341 

421 

0.20 

0.071 

0.401 

18 

328 

431 

0.15 

0.042 

0.344 

19 

318 

450 

0.10 

0.018 

0.283 

20 

298 

507 

0.05 

0.003 

0.216 
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5.  Confidence  Intervals 


Sonar  performance  measurement  is  a  statistical  sampling  process.  Therefore  it  is  not 
sufficient  in  this  example  to  simply  state  that  Pd=0.9  at  a  range  of  230m.  It  is  necessary 
to  construct  an  interval  about  tiie  Pd  and  with  a  specified  confidence  state  that  the  Pd 
lies  in  the  interval.  For  a  90%  confidence  interval,  the  statement  will  be  correct  90%  of 
the  time,  in  the  sense  that  if  we  repeated  the  experiment  many  times  and  computed  Pd 
and  the  corresponding  confidence  interval  for  each  of  them,  in  about  90  cases  out  of 
every  100  the  result  would  contain  Pd.  The  basic  distribution  is  binomial  and  the 
confidence  limits  are  tabulated  in  Table  21  and  charts  2,  3  and  4  of  Crow,  Davis  & 
Maxfield  [1].  The  two  sided  limits  should  be  used,  but  this  section  of  Table  21  in  Crow 
Davis  and  Maxfield  contains  several  errors  and  should  not  be  used.  The  one-sided 
limit  section  of  Table  21  has  been  checked  and  is  correct.  The  one-sided  limit  can  be 
used  to  derive  the  90%  two-sided  limit  by  taking  the  95%  value  from  the  table  as  the 
upper  limit.  The  lower  limit  is  obtained  by  entering  Table  21  with  n  and  n-r  and 
subtracting  the  table  entry  from  1.  For  example  if  n=20  and  r=18,  i.e.  a  Pd=0.9,  the 
upper  95%  one-sided  limit  is  0.982.  The  lower  95%  one-sided  limit  is  given  by  entering 
table  21  with  n=20  and  n-r=2.  This  gives  a  value  of  1-0.283=0.717.  Therefore  the 
measmed  Pd=0.9  and  the  90%  two-sided  confidence  limits  are  0.717  and  0.982.  The 
90%  two-sided  limits  for  values  of  n  up  to  30  have  been  obtained  by  this  method  and 
are  tabulated  in  the  Appendix.  A  copy  of  Chart  11  which  gives  confidence  limits  for 
larger  n  is  also  included  in  the  Appendix. 

Note  that  there  is  no  connection  between  the  Pd=0.9  value  and  the  90%  confidence 
intervals,  other  values  of  Pd  and  confidence  limits  could  have  been  chosen. 

It  is  important  to  consider  the  number  of  runs  to  be  carried  out.  If  the  number  is  small 
the  coittidence  interval  is  wide.  As  the  number  of  nms  increases  the  interval  reduces. 
The  following  table  shows  how  the  upper  and  lower  confidence  limits  vary  as  the 
number  of  runs  increases.  Values  are  shown  for  Pds  of  0.9  and  0.5. 

This  table  is  plotted  in  figure  1.  It  shows  clearly  that  for  small  n  the  confidence 
intervals  are  large.  For  example,  if  only  2  runs  are  carried  out  the  detection  range  for 
Pd=0.5  can  be  measured,  however  the  confidence  limits  on  the  Pd  at  that  range  are 
0.025  and  0.975.  If  250  runs  are  carried  out  the  limits  become  0.45  and  0.55.  A  Law  of 
Diminishing  Returns  is  operating  and  it  is  generally  considered  that  about  25  nms 
gives  a  smtable  balance  between  an  acceptable  confidence  interval  and  cost. 
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Table  2.  Number  of  Runs  and  Confidence  Intervals. 


Nmnber 
of  Runs 

Pd=0.5 

CL  lower 

Pd=0.5 

CL  upper 

Pd=0.9 

CL  lower 

Pd=0.9 

CL  upper 

2 

0.025 

0.975 

4 

0.098 

0.902 

6 

0.153 

0.847 

8 

0.193 

0.807 

10 

0.222 

0.778 

0.606 

0.995 

16 

0.279 

0.721 

20 

0.302 

0.698 

0.717 

0.982 

26 

0.327 

0.673 

30 

0.339 

0.661 

0.761 

0.972 

50 

0.380 

0.620 

0.820 

0.960 

100 

0.420 

0.580 

0.840 

0.940 

250 

0.450 

0.550 

0.870 

0.930 

1.0 
0.9 
0.8 
0.7 
0.6 
g  0.5 
0.4 
0.3 
0.2 
0.1 
0.0 


Pd=0.9- 


Rd=0.5 


0 


50 


90%  Confidence  Limit  (upper) 


T" 


90%  Confidence  Limit  (lower) 
90%  Confidence  Limit  (upper) 


90%  Confidence  Limit  (lower) 


+ 


100  150 

Number  of  Runs 


200 


250 


Figure  1.  90%  Confidence  limits  vs.  Number  of  runs. 
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The  upper  and  lower  90%  confidence  limits  for  the  example  data  are  shown  in  table  1 
and  are  plotted  in  figure  2.  This  figure  shows  the  area  within  which  the  90% 
confidence  limits  fall.  The  figure  can  be  interpreted  two  ways,  for  a  given  range  it 
shows  the  confidence  Hmits  on  the  probability  of  detection.  Alternatively  for  a  given 
probability  of  detection  it  shows  a  range  interval  over  which  that  Pd  appUes  with  a 
90%  confidence.  This  example  shows  that  the  Pd=0.9  at  a  range  of  230  m,  but  when  the 
90%  confidence  limits  are  considered  then  the  upper  confidence  limit  is  300  m.  Since  20 
detections  out  of  20  runs  (ie  Pd=1.0)  gives  a  lower  90%  confidence  limit  of  0.861  it  is 
not  possible  to  state  a  lower  range  at  which  tiiat  Pd  appHes. 

Section  2.3.3  of  Crow,  Davis  &  Maxfield  gives  a  numerical  solution  for  the  confidence 
limits  which  is  valid  for  large  samples.  It  is  approximately  true  for  the  sample  sizes 
encountered  in  sonar  evaluations  but  can  lead  to  values  of  Pd  just  greater  than  1  or  just 
less  than  0.  It  is  therefore  recommended  that  the  exact  tabulated  values  be  used. 


Figure  2.  Typical  minehunting  sonar  performance. 


6.  Specification  of  Performance  to  be  Demonstrated 

A  contract  between  the  sonar  supplier  and  the  cHent  Navy  should  state  that  the  sonar 
will  achieve  detection  at  a  given  range  with  a  given  probability  of  detection.  This 
performance  will  be  achieved  in  a  given  set  of  environmental  conditions  against  a 
target  of  a  specified  target  strength  and  with  a  specified  state  of  operator  alertness. 
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The  environmental  conditions  would  include  sound  speed  profile,  water  temperature, 
bottom  type,  wind  speed,  noise  level,  Pd  and  probability  of  false  alarm  (Pfa) . 

In  general  the  environmental  conditions  prevailing  during  the  sonar  trials  will  be 
different  to  those  specified  in  the  contract  and  will  influence  the  detection 
performance.  For  example  a  strongly  negative  sormd  speed  profile  may  severely  limit 
the  region  of  the  bottom  that  the  sonar  can  insonify,  and  therefore  limit  the  possible 
detection  range.  Obviously  some  method  must  be  used  to  compensate  for  this  t5^e  of 
situation  and  to  ensure  that  neither  side  in  the  contract  is  disadvantaged. 


7.  Use  of  Sonar  Model  to  Obtain  Predicted 
Performance 


A  sonar  model  is  used  to  predict  the  performance  of  the  minehimting  sonar.  The 
model  and  its  algorithms  need  to  be  agreed  between  the  Navy  and  the  Sonar 
Contractor.  The  inputs  needed  to  run  the  model  include: 

a.  Environment 

Sound  speed  profile 
Water  temperature 
Absorption 
Wind  speed 

Bottom  t5^e  or  measured  values  of  bottom 

Backscatter  strength 

Noise  levels  (  ambient,  flow,  machinery) 

Volume  scatter  strength 

b.  Sonar 

Source  level 

Beam  patterns  (vertical  &  horizontal,  transmit  and  receive) 

Frequency 
Pulse  length 
Type  of  transmission 
Sector  width 
Vertical  beams  and  tilt 
Processing  gains 

Transducer  depth  or  trail  back  (if  VDS) 

c.  Target 

Target  strength 
Dimensions 

The  model  uses  these  inputs  to  calculate  the  ratio  of  the  echo  to  the  noise  as  function  of 
range.  The  Receiver  Operating  Characteristic  (ROC)  is  used  to  obtain  the  S/N  for 
detection  for  a  given  value  of  Pd  and  Pfa.  Typically  Pd=0.9  and  Pfa=0.0001  give  a 
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S/N=+12  dB.  Some  relaxation  in  the  S/N  may  be  made  in  the  case  of  unindicated 
detections. 

During  the  trial  the  environment  is  regularly  monitored  and  the  model  is  used  to 
calculate  the  predicted  sonar  performance.  The  point  corresponding  to  the  predicted 
range  and  Pd  is  plotted  on  the  measured  Pd  curve.  Several  possible  outcomes  are 
shown  in  figure  3. 

In  fig  3a  the  measured  performance  is  better  than  predicted  since  the  predicted 
detection  range  falls  to  the  left  of  the  lower  confidence  limit  curve.  The  sonar  therefore 
satisfies  the  requirement  of  the  performance  specification. 

In  fig  3b  the  predicted  point  falls  between  the  upper  and  lower  confidence  limits.  The 
measurements  indicate  that  there  is  a  90%  chance  that  the  detection  range  for  the  given 
Pd  lies  between  the  upper  and  the  lower  confidence  limits.  In  this  case  there  is  some 
statistical  imcertainty  that  the  required  performance  has  been  met.  As  it  wiU  be 
difficult  to  resolve  this  uncertainty  one  way  or  the  other,  the  performance  wdl  be 
considered  to  have  passed.  If  however  the  predicted  point  falls  to  the  right  of  the 
upper  confidence  limit  curve  as  in  fig  3c  the  sonar  has  failed  to  meet  the  requirement 
of  the  performance  specification.  This  procedure  is  valid  only  if  the  enviroiunent 
remains  constant  dxuing  the  trial. 

If  there  are  significant  changes  in  the  environment  dxiring  the  trial,  then  for  each 
detection  range  there  wUl  be  a  corresponding  predicted  range.  Thus  the  trial  wiB 
generate  a  set  of  pairs  of  values  which  need  to  be  tested  to  determine  whether  the 
measured  values  are  better  or  worse  than  those  calculated  by  the  model.  A  suitable  test 
is  the  Wilcoxon  Signed  Ranks  Test  which  is  described  in  Siegel  &  Castellan  [2]. 


8.  Specification  of  Classification  Performance 

Classification  is  the  process  in  which,  once  a  mine  like  contact  is  detected,  the  operator 
determines  with  a  high  confidence  whether  it  is  a  mine  or  a  non-mine.  If  the  contact  is 
classified  as  a  mine,  it  will  be  furtitier  investigated  either  by  a  ROV  or  a  diver.  If  it  is 
classified  a  non-mine  no  further  action  is  required.  The  operator  uses  characteristics  of 
the  echo  such  as  shape,  length,  variation  with  aspect  and  shadow  to  classify. 

As  the  object  is  either  a  mine  or  a  non-mine,  and  the  operator  can  classify  as  a  mine  or 
non-mine,  there  are  4  outcomes  which  constitute  a  decision  matrix. 

Table  3.  Classification  decision  matrix. 


OBJECT  IS 

CLASSIFY  AS 

MINE 

NON-MINE 

MINE 

Pccm 

1-Pccnm 

NON-MINE 

1-Pccm 

Pccnm 
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Range  (metres) 

Figure  3A.  Predicted  range  less  than  measured  range. 


0  100  200  300  400  500  600 


Range  (metres) 

Figure  3B.  Predicted  range  greater  than  measured  range  but  less  than  ujrper  limit. 


0  100  200  300  400  500  600 

Range  (metres) 

Figure  3C.  Predicted  range  greater  than  upper  confidence  limit. 

Figure  3.  Comparison  between  measured  and  specified  performance. 
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If  the  object  is  a  mine  then  the  probability  of  correctly  classifying  it  as  a  mine  is  Pccm. 
If  the  object  is  a  mine  and  is  incorrectly  classified  as  a  non-mine  the  associated 
probability  is  1-Pccm.  Similarly  if  the  object  is  a  non-mine  then  the  probability  of 
correctly  classifying  it  as  a  non-mine  is  Pccnm  The  probability  of  classifying  a  non¬ 
mine  as  a  mine  is  1-Pccnm.  There  is  a  different  penalty  associated  with  each  of  these 
wrong  decisions.  If  a  non-mine  is  classified  as  a  mine,  time  is  wasted  in  the 
investigation.  However,  if  a  mine  is  classified  as  a  non-mine  the  minehimter  and  other 
ships  may  be  placed  at  risk. 

The  classification  performance  to  be  satisfied  is  specified  by  these  two  probabilities, 
Pccm  and  Pccnm. 


9.  Measurement  of  Classification  Performance 


Classification  performance  is  determined  by  approaching  an  object  until  a  detection  is 
made.  The  operator  is  then  free  to  manoeuvre  the  ship  to  vary  the  aspect  and  to 
approach  to  a  minimmn  range,  corresponding  to  the  safety  range.  The  operator  is  then 
required  to  classify  as  either  a  mine  or  a  non-mine.  If  the  object  is  a  mine  then  Pccm  is 
given  by  the  number  of  correct  classifications  divided  by  the  number  of  approaches 
made  to  the  mine.  For  example,  if  25  approaches  were  made  to  a  mine  and  the  operator 
classified  it  as  a  mine  on  20  occasions  the  Pccm=20/ 25=0.8.  Similarly  if  the  object  is  a 
non-  mine  then  Pccnm  is  given  by  the  munber  of  correct  classifications  divided  by  the 
mimbpT  of  approaches  made  to  the  non-mine. 

In  order  to  measure  classification  performance  a  number  of  mines  and  non-mines  are 
laid.  The  mines  and  non-mines  should  be  encountered  at  random  so  that  no  pattern  is 
discernible  to  the  operator.  The  feahires  of  the  non-mines  should  be  carefully  selected 
to  enable  the  classification  sonar  to  distinguish  them  from  mines.  For  example  the 
mines  could  consist  of  two  cylindrical  grotmd  mines  (length=2.0m,  diameter=0.5m) 
and  (length=1.5m,  diameter=0.3m)  and  a  moored  mine  with  cable  and  sinker.  The 
non-mines  could  consist  of  a  fluid  filled  target  sphere  of  diameter  15cm,  a  length  of 
free  flooding  cylindrical  water  pipe  with  end  caps  (length  3.0m,  diameter  0.5m)  and  an 
approximately  spherical  rock  of  diameter  0.6m.  The  operators  would  be  briefed  on  the 
nature  of  the  threat  mines  before  the  trial.  The  sphere  should  be  distinguishable  from 
the  mines  because  of  its  size.  Similarly  the  classification  sonar  should  be  able  to 
measure  the  length  of  the  pipe  and  classify  it  as  a  non-mine.  The  shape  of  the  rock 
plus  any  shadow  shotdd  enable  it  to  be  classified  as  a  non-mine. 

It  is  important  to  have  approximately  equal  numbers  of  mines  and  non-mines.  If  there 
are  9  mines  and  1  non-mine  then  by  classifying  everything  as  a  mine  Pccm  would  be 
estimated  to  be  0.9,  an  apparent  good  result. 
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In  a  similar  manner  to  the  detection  performance  calculations  confidence  limits  can  be 
obtained.  If  25  runs  are  carried  out  then  the  following  table  shows  Pccm,  n  the  number 
of  correct  classifications  and  the  upper  and  lower  90%  confidence  limits. 


Table  4.  90%  Confidence  limits  for  n  correct  classifications  in  25  instances. 


n 

Pccm 

CLlower 

CLupper 

19 

0.76 

0.610 

0.892 

20 

0.8 

0.638 

0.899 

21 

0.84 

0.693 

0.929 

22 

0.88 

0.745 

0.955 

23 

0.92 

0.786 

0.979 

24 

0.96 

0.841 

0.996 

25 

1.00 

0.898 

1.000 

In  order  to  not  rule  out  a  Pccm  of  0.9  then  there  must  be  at  least  20  correct 
classifications  out  of  25. 


10.  Conclusion 


A  methodology  has  been  described  which  enables  the  detection  and  classification 
performance  of  a  minehunting  sonar  to  be  rigorously  measured.  Confidence  hmits  can 
also  be  placed  on  the  measurements.  A  method  of  using  a  computer  model  to  allow  for 
the  effects  of  the  environment  is  also  described.  The  procedures  are  suitable  for  use  as 
contract  acceptance  trials. 
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Appendix 


CHART  II.  90%  Confidence  Belts  for  Proportions 


*U8e  explained  in  Sec.  2.3.3&.  Ta.ble  21  should  be  uned  for  sample  size  n'^^-SO.  It  ariven 
the  interval  explicitly  for  each  sample  size.  This  chart  was  adapted  with  permission  from  W. 
J.  Dixon  and  F.  J.  Massey,  Jr.,  Introduetion  to  SlatiMtiral  Analytrin.  New  York.  MeGr4w> 
Hill.  1951.  p.  320. 


Fig  Al.  90%  Confidence  limits  for  a  proportion  for  n>30. 
(Extract  from  Crow,  Davis  and  Maxfield) 
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Table  Al.  90%  Confidence  Limits  Jbr  a  Proportion  (continued). 
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